A typical digital camera uses a single color image sensor to capture a representation of a scene, including a representation of the color in the scene. The colors are separated by a color filter array(CFA) that overlies the sensor and associates each image pixel with a particular color according to the particular color pattern of the CFA. For example, the Bayer CFA (described in U.S. Pat. No. 3,971,065) provides an RGB pattern with a checkerboard arrangement of green pixel sites. Therefore, the primary CFA image signal for each pixel relates to but one color, that is, to either red, green or blue (for a Bayer CFA). After the CFA image is captured, the remaining (missing) colors for each pixel site are interpolated for each pixel from, e.g., surrounding pixels, so that a full resolution interpolated record is generated for each image. Each pixel in the interpolated record therefore has a set of RGB values.
Digital cameras, such as the Kodak DC50 camera, store the compressed digital image data from a single-chip color sensor (approximately 750×500 pixels) on a removable memory card. The process used to convert the image to a standard file format, and to print and edit the standard format, is shown in FIG. 1. The CFA image is captured in a capture step 10 by the camera. When the camera or memory is connected to a host computer in a connect step 12, the images are downloaded from the camera or memory card in a downloading step 14, processed in a processing step 16 and stored in a standard file format, such as a FlashPix file format, in a file formatting step 18. In the processing step 16, the CFA images are CFA interpolated as they are imported, using known CFA interpolation, color correction, and sharpening techniques, in order to create a “finished” RGB image file having a standard size. The finished RGB image file thus contains the CFA interpolated image, which has the same number of total pixels as the original image (for example, approximately 750×500 pixels for the DC50 camera). Unlike the original image, however, the CFA interpolated image has complete RGB data for each pixel.
The advantage of using a standard file format is that it allows the image to be used by many different imaging-enabled programs supplied by many different companies. Many different file formats are known and useful with this invention, albeit the following description is mainly in terms of a FlashPix file. FIG. 2 is a simplified diagram showing the key information stored in a FlashPix file. The FlashPix image format (defined in FlashPix Format Specification, version 1.0.1, (Digital Imaging Group, Jul. 10, 1997)) has been developed to serve as both an “interchange” format between devices (e.g. cameras) and applications (e.g. computer picture editing packages), and as a “native” format for image editing that allows the images to be easily and rapidly edited. This is accomplished using a hierarchical, tiled image representation within a “structured storage” file. Referring to FIG. 2, a FlashPix file contains the complete image data 24 plus a hierarchy of several lower-resolution copies within the same file (one set of lower resolution image data 25 is shown in FIG. 2).
Images at each resolution also are divided into rectangular tiles (e.g., squares), which enable the application to minimize the amount of image data processed to access, display or print a portion of the scene content. FlashPix allows thumbnail image data 23 and optional “ancillary” property set data 21 and 22 to be stored in the same structured storage file, along with the image data. This ancillary data can include digital audio recordings and parameters such as the time and date the photo was taken, the camera zoom position and focus distance, the scene illumination level, camera calibration data, image copyright owner, etc. For further information about the FlashPix image format see the aforementioned FlashPix Format Specification, version 1.0.1, (Digital Imaging Group, Jul. 10, 1997), which is available on the Wide World Web at http://www.digitalimaging.org/go/flashpix, and is incorporated herein by reference.
The image processing used to “finish” the image prior to storing the finished data in the FlashPix file includes operations such as CFA interpolation, color correction, and image sharpening. The output images are normally stored in a standard color space, such as the sRGB space which is supported by the FlashPix image format. As a consequence of generating the image file, the original camera data is discarded (step 26). If an image is to be printed or transmitted in an application step 28, the standard file format is first edited in an editing step 27.
To create final printed images, the finished image file may be adjusted by the user via an image processing program, such as provided by a program like LivePixp™ by Live Picture, Inc., to create a final output image of the desired size, which may include only a “cropped”, color adjusted portion of the image captured by the camera, or may, e.g., include multiple images in a “montage”. More specifically, to provide the appropriately sized output image, the image processing program first crops the 750×500 RGB CFA interpolated image data, and then uses another interpolation algorithm in the editing step 27 to convert from the cropped version of the “finished” RGB image file to the final desired output image.
Note that this prior art approach uses two interpolation steps, one (step 16) to interpolate “missing” RGB data from the image sensor while maintaining the 750×500 pixel data array, and a second (step 27) to interpolate from the selected crop to the pixel data array that provides the desired output size. The user therefore selects the final print size (for example 8″×10″), and the image is interpolated up to produce an image data record of appropriate size for the printer. For example, if the output device printed at 200 pixels per inch, the image data would be interpolated to 1600×2000 pixels by the computer or the printer, (and optionally sharpened) to produce the desired image size. In addition, this known approach often uses two separate sharpening operations, one on the 750×500 pixel CFA interpolated image data following the CFA interpolation step 16, and a second one on the interpolated 1600×2000 pixel image data array just prior to printing.
The prior art approach has a number of problems. First, the process of converting the image from the CFA image data (which may for example produce 12 bit per color data) to the 8 bit per color (24 bit per pixel) sRGB color space may limit the quality of certain images, by reducing the color gamut and dynamic range of the captured image. Second, the two interpolation steps (the CFA interpolation that produces a standard size image and the interpolation in the computer or printer that produces the desired print size) provide more interpolation artifacts than would be produced using a single interpolation step that interpolates from the sensor CFA data directly to the output image data sent to the printer. Finally, using two separate sharpening steps can also produce artifacts.
What is needed is a digital camera system that maintains the advantages of using a standard, “finished” image file format so that the images can be used by many applications, yet also enables image processing from raw camera data to final output data to be completed in a single, integrated process, to provide improved image quality when printing.